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Assistant Commissioner of Patents 
Washington, DC. 20231 



Dear Sir: 

FmST PRELIMINARY AMENDMENT 

Please enter the following preliminary amendment to the application: 

Replacement Specification 

Please delete the specification currently on file and replace with new specification, pages 
1 through 19, as submitted in Attachment A. 

In the Disclosure 

Please add new paragraph 000 as submitted in Attachment B. 
In the Drawings 

Please replace Figures 1, 2, 3a, 3b, 3c, 3d, 3e, 4a, 4b, 4c, 4d, 5, 6, 7, 8, 9 and 10 currently 
on file with new Figures 1, 2, 3a, 3b, 3c, 3d, 3e, 4a, 4b, 4c, 4d, 5, 6, 7, 8, 9 and 10 as submitted in 
Attachment A. 

Attachment A is a replacement specification, claims, abstract and formal drawings. 
Attachment B is a clean copy of the amendment. 



Attachment C is a marked up copy of the amendment. 



REMARKS 



To facilitate the examination process, a replacement specification, claims, abstract and 
drawings is provided as Attachment A. No new matter has been added. This document is 
identical to the international publication number WO 01/22149 Al published on March 29, 2001 . 

We request a statement be added at the beginning of the specification under 35 U.S.C. 
1 19(e) and in accordance with 37 CFR 1.78. A clean copy of the new paragraph and a marked up 
copy of the new paragraph is attached in Attachment B and Attachment C. 

Figures 1, 2, 3a, 3b, 3c, 3d, 3e, 4a, 4b, 4c, 4d, 5, 7, 8, 9 and 10 include computer- 
generated images. As computer-generated images, they are helpful in demonstrating the 
exemplary computer implemented process of the method of the invention. 

Applicant looks forward to receiving the first Official Action from the Examiner in due 

course. 

Respectfully submitted, 




Gordon Freedman 
Reg. No. 41,553 



Freedman & Associates 
117 Centrepointe Drive 
Suite 350 
Nepean, Ontario 
Canada K2G 5X3 



Tel (613)274-7272 
Fax (613)274-7414 
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ATTACHMENT B 

Clean New Paragraph 



After the Title 

Page 1, please insert new paragraph 000 directly after the title: 

[000] This application is a national phase entry of International Application No. 
PCT/CA00/01063 filed September 22, 2000, which claims priority from United States 
Provisional Patent Application No. 60/155,132 filed September 22, 1999, now abandoned. 
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ATTACHMENT C 

Marked Up New Paragraph 

After the Title 

Page 1, please insert new paragraph 000 directly after the title: 

[0001 This application is a national phase entry of International Application No. 
PCT/CA00/O1063 filed September 22. 2000. which claims priority from United States 
Provisional Patent Application No. 60/155.132 filed September 22. 1999. now abandoned. 
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D 
g 

go 



M 
n 



3 



Doc. No. 50412-01 US/PCT 



c 



10/ 0887^7 

x% \\^ JC13 B8C ? d PCT7PTC 2 2 MAR 2002 



Method and System for Time/Motion Compensation 
for Head Mounted Displays 

Field of the Invention 

[001] The present invention generally relates to telepresence systems and more 
particularly relates to motion compensation in telepresence systems. 

Background of the Invention 

[002] The field of remote control has come a long way since the days of watching a 
model aircraft fly under the control of a handheld controller. Robotics and remote robotic 
manipulation have created a strong and pressing need for more remote and better remote 
control systems. Obviously, an ideal form of remote control involves providing an 
operator with all the sensations of operating the remote robot without the inherent 
dangers, travel, and so forth. In order to achieve this, a telepresence system is used. 

[003] Telepresence systems are sensory feedback systems for allowing sensing and 
monitoring of remote systems. A typical telepresence sensor is a camera and a head 
mounted display. The system provides visual feedback from a remote location to an 
operator. For example, in a telepresence system for an automobile, the front windshield is 
provided with a camera. The controls of the vehicle are provided with actuators for 
automatically manipulating same. An operator is provided with a duplicate of the cabin of 
the car. The windshield is replaced with a display and the controls are linked via 
communications to the actuators within the vehicle. Turning of the steering wheel in the 
cabin of the car causes the steering wheel to turn in the vehicle. Similarly, the camera 
captures images in front of the car and they are displayed on the display in the cabin of 
the car. 

[004] Presently, there is a trend toward providing the visual feedback using a head 
mounted display (HMD). A head mounted display is a small display or two small displays 
mounted for being worn on a users head. Advantageously, an HMD with two displays 
provides stereo imaging allowing a user to perceive depth of field. Alternatively, such an 
HMD provides two identical images, one to each display. Unfortunately, the head 
mounted display only presents a user with information from approximately in front of the 
user. Thus, when a user turns their head, the image seen and the expected image differ. 



1 



Doc. No. 50412-01 US/PCT 

Therefore, the camera is mounted on a mechanism which moves in accordance with 
detected HMD movement. Thus, the image before the user is in accordance with the 
user's head position. 

[005] Generally, it is an object of telepresence systems to provide a visual sensation 
of being in the place of the robot and a control system for controlling the robot as well. 
Thus, telepresence systems aim to provide feedback that is appropriate to different 
situations. 

[006] Unfortunately, a camera does not move in exact synchronisation with the 
HMD so the image is not perfectly aligned with the expectations of the user during head 
motion. This misalignment can result in disorientation and nausea on the part of an 
operator. 

[007] The disclosures in U.S. Patent 5,579,026 issued on Nov. 26, 1996 in the name 
of Tabata and in U.S. Patent 5,917,460 issued on Jun. 29, 1999 in the name of Kodama 
focus on image display for use in, for example, virtual reality and games. In there is 
described a head mounted display in which the position of the projected image can be 
displaced in response to a control unit or in response to the rotational motion of the 
operator's head. The essence of the head-tracking implementation is that from the user's 
perspective, the image can be made to remain substantially stationary in space during 
head movements, by being manipulated in a manner opposite to the movements. 
Significantly, the patents do not relate to visual telepresence using slaved cameras. In the 
slaved camera implementation, the camera should follow the motion of the HMD and, as 
such, compensation for HMD motion is unnecessary since the image is always of a 
direction in which the head is directed. 

[008] Further because U.S. Patent 5,579,026 relates to displaying a simulated planar 
image, such as a simulation of a television screen located in virtual space in front of the 
user, the patent provides for a fixed frame of reference relative to a wearer of the HMD. 
The images in any direction are simulated thus being formed as needed. Unfortunately, in 
telepresence systems, often the video data relating to a particular direction of view is 
unavailable. This complicates the system significantly and as such, the prior art relating to 
video data display is not truly applicable and, one of skill in the art would not refer to 
such. 
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[009] In U.S. Patent 5,917,460 issued on Jun. 29, 1999 in the name of Kodama a 
system addressing the three-axes displacement (up/down, left/right, 
frontwards/backwards) of a HMD is provided. The displacement appears to linear and is 
accommodated through a mechanical mechanism. The displays are moved in response to 
detected movement of a head and as such, objects remain somewhat stationary from the 
visual perspective of the user. 

[0010] It is not well suited to use in telepresence wherein a camera tracks the motion 
of the HMD. One of skill in the art, absent hindsight, would not be drawn to maintaining 
a visual reference when a head is turned, for a telepresence system wherein a camera is 
rotated in response to head movement. Of course, the different problem results in a 
different solution. 

[0011] For example, in telepresence systems, the delay between camera image 
capture and head motion is often indeterminate. It is not a workable solution to implement 
the system of the above referenced patents to solve this problem. Because of the unknown 
delays caused by camera response time and communication delays, the solution is not 
trivial. 

[0012] In U.S. Patent 5,933,125 a system is disclosed using prediction of the head 
movement to pre-compensate for the delay expected in the generation of a virtual image, 
nominally in a simulated environment. By this means, a time lag in the generation of 
imagery is compensated for by shifting the scene to provide a stable visual frame of 
reference. This method is applicable to short delays and small displacements, where head 
tracking information can be used to predict the next head position with reasonable 
accuracy. The patent discloses 100msec as a normal value. Effective prediction of head 
motion is aided by comprehensive information about head movement, including angular 
head velocity and angular acceleration. For small head movements, errors induced are 
small. Typically, these occur in a small period of time. The disclosed embodiments rely 
on knowledge of the time delay, which is nominally considered to be constant. 
Unfortunately, when the time delays grow large allowing for substantial motion of a head, 
the errors in the predictive algorithm are unknown and the system is somewhat 
unworkable. 
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[0013] Furthermore, US 5,933,125 cannot compensate for unanticipated image 
movement, only that which occurs in correct response to the operator's head movement. 
Also, it does not relate to visual telepresence systems using remote slave cameras. 

[0014] It would be highly advantageous to provide a system that does not rely on any 
form of prediction for compensation and which works with variable delays between 
image capture and image display. 

Object of the Invention 

[0015] In order to overcome these and other shortcomings of the prior art, it is an 
object of the invention to provide a method of compensating for time delays between 
head motion and camera motion in telepresence systems. 

Summary of the Invention 

[0016] The invention relates to a method and apparatus that provides a wearer of an 
HMD with a stable frame of visual reference in cases where there may be time delays or 
unwanted motion within the visual capture/visual display systems. 

[0017] According to the invention, in order to eliminate some of the disorientation 
caused by time delays in camera motion when a head motion occurs, an image shown on 
the display of a head-mounted display (HMD) is offset relative to the field of view of the 
HMD until the camera position is again synchronised with the HMD position. Offsetting 
of the image results in areas of the display for which no image information is available. 
These display areas are provided fill data in the form of a solid shading or some feature 
set for providing visual cues. When the transformed images again overlap the display, the 
fill is no longer necessary. 

[0018] In accordance with the invention there is provided a method of motion 
compensation for head mounted displays. The method includes the following steps: 
providing an image from an image capture device to a head mounted display including a 
monitor having a field of view; providing camera position data associated with the image; 
providing head position data; adjusting the image location relative to the field of view of 
the monitor in accordance with the camera position data and the head position data; and, 
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displaying portions of the image at the adjusted locations, those portions remaining within 
the field of view. 

[0019] Typically position data includes at least one of orientation data and location 
data. Location data is also referred to as displacement data. Typically, portions of the 
field of view without image data are filled with a predetermined fill. When none of the 
image data is for display within the field of view, the entire field of view is filled with the 
predetermined fill. 

[0020] For example, the image is adjusted by the following steps: determining an 
offset between the head mounted display position and the camera position; and, offsetting 
the image such that it is offset an amount equal to the offset between the head mounted 
display position and the camera position. 

[0021] Advantageously, such a system is not limited by the accuracy of a predictive 
process nor by the time delay between image capture and image display. Instead, it is 
reactive, and uses sensed information on HMD position and camera position to formulate 
a transformation for the captured image. The present invention has no limit to the time 
delays for which compensation is possible since the required head position information 
and camera position information are sensed at different times allowing compensation for 
any delay between sensing one and then sensing the other. 

[0022] Further advantageously, the present invention requires no knowledge of the 
time delay in the system and functions properly in the presence of non-constant time 
delays. There is no requirement that the time delay be measured and it is not used in 
determining the transform of the image. 

Brief Description of the Drawings 

[0023] The invention will now be described in conjunction with the drawings in 
which: 

[0024] Fig. 1 is a simplified block diagram of a system incorporating an HMD 
coupled to a first computer and in communication with a remote camera; 

[0025] Fig. 2 is a simplified diagram showing axes of movement of the systems 
involved; 
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[0026] Fig. 3 a is a simplified diagram showing a simulated view of an image 
appearing within the field of view of an HMD; 

[0027] Fig. 3 b is a simplified diagram showing a simulated view of a portion of the 
image of Fig. 3a offset vertically within the field of view of an HMD in response to a 
downward motion of a user's head; 

[0028] Fig. 3c is a simplified diagram showing a simulated view of a portion of the 
image of Fig. 3 a offset horizontally within the field of view of an HMD in response to a 
lateral motion of a user's head; 

[0029] Fig. 3d is a simplified diagram showing a simulated view of a portion of the 
image of Fig. 3 a tilted within the field of view of an HMD in response to a tilting motion 
of a user's head; 

[0030] Fig. 3e is a simplified diagram showing a simulated view of a portion of the 
image of Fig. 3a tilted and offset both vertically and horizontally within the field of view 
of an HMD in response to a tilting motion combined with a lateral and a horizontal 
motion of a user's head; 

[0031] Fig. 4a is a simplified diagram showing a simulated view of an image 
appearing within the field of view of an HMD; 

[0032] Fig. 4b is a simplified diagram showing a simulated view of a portion of the 
image of Fig. 4a offset horizontally within the field of view of an HMD in response to a 
lateral motion of a user's head; 

[0033] Fig. 4c is a simplified diagram showing a simulated view of an image 
appearing within the field of view of an HMD including a portion of the image of Fig. 4a 
as well as additional image data captured and displayed within the field of view of an 
HMD when the camera motion is partially caught up with the lateral motion of a user's 
head; 

[0034] Fig. 4d is a simplified diagram showing a simulated view of an image 
appearing within the field of view of an HMD including a portion of the image of Fig. 4a 
as well as additional image data captured and displayed within the field of view of an 
HMD after the camera motion is fully caught up with the lateral motion of a user's head; 
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[0035] Fig. 5 is a simplified block diagram of a system incorporating an HMD 
coupled to a first computer and in communication across a network with a second 
computer coupled to a remote camera; 

[0036] Fig. 6 is a simplified flow diagram of a method according to the invention; 

[0037] Fig. 7 is a simplified block diagram of a telepresence system communicating 
via a satellite communications link; 

[0038] Fig. 8 is an image captured by a remote camera as captured; 

[0039] Fig. 9 is an image of the same image as that of fig. 8 transformed within an 
endless image display space; and, 

[0040] Fig. 10 is an image of the same image as that of fig. 9 as displayed on a 
display having a finite image display space. 

Detailed Description of the Invention 

[0041] The present invention is described with reference to telepresence systems 
operating over long distances such that significant time delays occur between head 
motion and image display of an image for a current head orientation. It is, however, 
equally applicable when a camera drive mechanism provides insufficient response rate to 
allow comfortable viewing of images during normal head motion. It is also applicable in 
situations where unwanted and unmodeled motion of the camera is possible, such as when 
the camera is mounted on a moving platform. 

[0042] Referring to Fig. 1, a simplified block diagram of a telepresence system is 
shown. A head mounted display (HMD) 1 including a three-axes head tracker 3 is worn 
by an operator 5. The HMD 1 is coupled with a first computer 7 and provides to the first 
computer 7 HMD values for the HMD position in the form of pitch 21, yaw 22, and roll 
23 angles of the HMD 1 as shown in Fig. 2. Of course, since the HMD 1 is being worn by 
an operator 5, these HMD values relate to the head position of the operator 5. These 
values are provided to the first computer 7 at intervals, preferably more than 100 times 
each second, though other intervals are also acceptable. The HMD values are converted 
by the first computer 7 into control values for controlling positioning of a camera 1 1 . The 
control values are transmitted to a mechanism 13 for pointing the camera in order to 
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affect camera orientation. As is seen in Figure 2, the mechanism controls pitch 24, yaw 25 
and roll 26 of the camera 1 1 . In response to the control values, the camera position moves 
in accordance with the movement of the HMD 1 . 

[0043] When the mechanism 1 3 for pointing the camera is physically coupled to the 
first computer 7, the camera 1 1 begins to move when HMD motion is detected. The lag 
between camera motion and HMD motion is determined by communication delays, which 
are very small, processing delays, which may be minimal, and pointing mechanism 
performance, which varies. These delays often result in an image provided from the 
camera 1 1 remaining static while the HMD 1 is in motion or moving substantially slower 
than the HMD motion. Of course, since the operator's mind expects motion within the 
visual frame, this is disconcerting and often results in nausea and disorientation. 

[0044] This problem is even more notable when communication delay times are 
significant such as when used for terrestrial control of systems in space. There, the delay 
is in the order of seconds and, as such, the disorientation of an operator during HMD 
motion is significant. Significantly, disorientation is a cause of operator fatigue resulting 
in limited operator use of a system or limited use of a system during a day. 

[0045] Referring again to Fig. 1, the camera 1 1 is constantly acquiring images at a 
video data capture rate. Each image is transmitted to the first computer for processing, if 
required, and for provision to the HMD 1. According to the invention, the remote system 
also provides camera position information to the first computer 7 and associated with 
each image. Thus, each frame received by the first computer 7 has associated therewith 
camera position information. The camera position information is preferably relative to a 
known orientation. Alternatively, it is transformed by the first computer 7 into position 
information relative to a known camera orientation and in a coordinate space analogous to 
that of the HMD 1. 

[0046] The HMD position values are used to determine a current HMD orientation in 
a coordinate space analogous to that of the camera 1 1 . As such, an offset between camera 
orientation and HMD orientation is determinable. Since the HMD 1 is being worn by an 
operator 5 the HMD orientation is directly correlated to the position of the head of the 
operator 5. Of course, the direct correlation is related to sensed position data and in use is 
generally an approximate direct correlation due to a refresh rate of the HMD position 



Doc. No. 50412-01 US/PCT 



sensor. The offset between the camera orientation and the HMD orientation is related to a 
delay between the local system and the remote system. 

[0047] Therefore when a non-zero offset is determined, the first computer offsets the 
image provided by the camera relative to the field of view of the HMD in order to 
compensate for the determined offset. Referring to Figs. 3a, 3b, 3c, 3d and 3e, some 
examples of image displays are shown. In Fig. 3a, the image is shown for a zero offset 
between camera orientation and HMD orientation. This is the steady state of the feedback 
system since the HMD 1 is directed in a same direction as the camera 1 1 and the image 
displayed on the display within the HMD 1 is the same as the image captured by the 
camera. When the HMD orientation is angled down from the camera position, the image 
is offset in a vertical direction as shown in Fig. 3b. When the camera orientation is offset 
horizontally, the image is offset horizontally as shown in Fig. 3c. In Figs. 3d and Fig. 3e, 
the image is rotated and offset relative to the field of view because the camera orientation 
is rotated and offset relative to the HMD orientation. 

[0048] Though the described instantaneous corrections shown in Figs. 4a, 4b, 4c and 
4d appear simple, the steady state nature of the system requires an ever changing imaging 
perspective and display perspective. Thus a comparison is necessary between two 
dynamic sets of sensed position data. 

[0049] Referring to Figs. 4a, 4b, 4c and 4d, field of view is shown for the HMD 1 
during a left turn of the operator's head. At first (before the head turn) in a steady state, 
the exact image captured by the camera 1 1 is shown in the display of the HMD 1 at Fig. 
4a. When the head turns, the operator 5 "expects" the image to move to the right since the 
image is not part of the operator 5 and is within their field of view. This expectation is 
either conscious or unconscious. Imagining that the image remains static as the HMD 
moves, it is clear that disorientation would result since individuals take cues from their 
visual field of view during head movement. In order to provide the operator 5 with the 
"expected" displacement of objects in the image, the image is offset to a location 
approximately the same as the orientation difference between the HMD 1 and the camera 
11. For example, in the image of Fig. 4b the lighthouse is shifted out of the field of view 
by the rotation of the head. Turning the HMD a degrees, an operator expects static 
objects within the field of view, such as a lighthouse, to shift a degrees within the field of 
view. This is important to maintaining comfort of the operator in their personal vision 
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system (their eyes and their mind). At a same time, the camera begins to move to match 
its orientation to that of the HMD. Thus, as shown in the image of Fig. 4c more of the 
scene within the operator's field of view is now available from the camera 1 1. As the 
camera orientation "catches up" with the HMD orientation, the field of view of the 
camera and that of the HMD overlap more. When the camera 1 1 is "caught up," the field 
of view of the HMD again shows an entire image captured by the camera as shown in the 
image of Fig. 4d. 

[0050] Referring to Fig. 5, another embodiment of the invention is shown for use on a 
network. Here for example, two computers 7 and 15 communicate via a network or 
networks 17. The first computer 7 includes the HMD 1 as a peripheral thereof. The 
second computer 15 includes the camera 1 1 and mechanism 13 for pointing the camera as 
peripherals thereof. Here the processing is performed in either of the first computer 7 or 
the second computer 1 5 though the image processing is preferably performed in the first 
p computer 7 in case of network delays that could cause image offset and result in 

: - disorientation of an operator 5. Of course, when network delays are known to be 

tv 

Sj significant, it is important that image processing is performed on the first computer. 

*P 

[0051] Referring to Fig. 6, a simplified flow diagram of a method of performing the 
p invention is shown. An image is captured by the camera 1 1 . A sensor captures position 

data in the form of camera orientation values for pitch, roll and yaw. The position data is 

FU preferably captured concurrently with the image. Alternatively, it is captured 

trl 

r approximately at a same time but offset by a finite amount either immediately after image 

rtl 

capture or immediately before. The position data is then associated with the image data. A 
simple method of associating the data is by encoding the position data with the image 
data either as header or trailer information. Of course, the image and the position data 
could also be identified with an associating identifier such as an image frame number. 
Alternatively, the two data are transmitted in parallel in a synchronous environment. 

[0052] The image and position data are then transmitted to the first computer 7. When 
the image and position data are received, they are prepared for processing at the first 
computer 7. Then, the position data of the HMD 1 is acquired by the first computer 7 and 
is used to transform the image in accordance with the invention. The transformed image 
is provided to the display and is displayed thereon to the operator 5. Because the HMD 
position data is gathered immediately before it is needed, the delay between HMD 
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position data capture and display of the transformed image is very small and results in 
little or no operator disorientation. 

[0053] Concurrently, position data is provided to the mechanism 13 at intervals and 
the mechanism moves the camera 1 1 in accordance with received position data and a 
current orientation of the camera 1 1 . 

[0054] Typically, the step of transforming the image comprises the following steps, 
some of which are performed in advance. A correlation between angular movement and 
display or image pixels is determined such that an offset of a degrees results in 
displacement of the image by N pixels in a first direction and by M pixels in a second 
other direction. A transform for rotating the image based on rotations is also determined. 
Preferably, the transforms are sufficiently simple to provide fast image processing. That 
said, a small image processing delay, because it forms substantially the delay in 
displaying the data, is acceptable. 

[0055] Once the image data is received, it is stored in memory for fast processing 
thereof. The HMD position data is acquired and is compared to the camera position data. 
The difference is used in performing the transform to correct the image position for any 
HMD motion unaccounted for by the mechanism 13, as of yet. Also, the method corrects 
for unintentional movements of the camera 1 1 when the camera position sensor is 
independent of the mechanism 13, for example with an inertial position sensor. 

[0056] In the above embodiment, a general purpose processor is used to transform the 
image. In an alternative embodiment, a hardware implementation of the transform is used. 
A hardware implementation is less easily modified, but has a tremendous impact on 
performance. Using parallel hardware transformation processors, an image can be 
transformed in a small fraction of the time necessary for performing a software 
transformation of the image. 

[0057] Referring to Fig. 7 a satellite based telepresence system is shown. Here delays 
in the order of seconds between head motion and image display result. Further, the delays 
are not always predictable. Here an HMD 101 is shown positioned on the head of an 
operator 5. The HMD is provided with a head tracker 103 for sensing position data 
relative to the HMD. The HMD is also coupled with a computer 1 07 for providing display 
data to the HMD and for providing the HMD position data to a communications link 108. 
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The communications link 108 uplinks the HMD position data to a satellite 117 from 
which it is transmitted to a transceiver 1 16. A computer 1 15 in communication with the 
transceiver provides the data to a gimbal 1 13 for repositioning a camera 1 1 1 in 
accordance therewith. The camera 1 1 1 captures images which are provided along a 
reverse communication path - computer 115, transceiver 116, satellite 117, 
communications link 108 - to the computer 107. Optionally, a different return path is 
used. There the image data is processed for display within the HMD 101. With the 
images, camera position data sensed by a sensor 1 14 is also provided. The camera 
position data is associated with an image or images captured at approximately a time the 
camera 1 1 1 was in that sensed position. 

[0058] The computer 1 07 uses the camera position data and the image along with data 
received from the head tracker 103 to transform the image in accordance with the 

M 1 invention. As is evident, the delay between HMD motion and camera motion is 

Q 

p measurable in seconds. The delay between camera image capture and receipt of the image 

at the computer 107 is also measurable in seconds. As such, significant disorientation of 
the user results absent application of the present invention. 

^ [0059] Referring to Fig. 8, an image captured by the camera 1 1 1 is shown. The image 

qj is displayed as captured when the HMD and the camera orientations are aligned, the 

camera orientation at a time of image capture and the HMD orientation at a time of 
fj display. If the orientations are offset one from another, the image is shifted within the 

field of view of the operator as shown in Fig. 9. Since there is no image data beyond the 
camera imaging field of view, the remainder of the display area is shaded with a neutral 
colour such as gray. Alternatively, the remainder of the display area is shaded to provide 
reference points to further limit disorientation of the operator 105. Further alternatively, 
the portion of the field of view for which no image data is available is left blank. 
Typically blank areas are black in order not to distract an operator. Referring to Fig. 10, 
when the camera 1 1 1 orientation is "caught up" with the HMD orientation, the field of 
view of the HMD again shows an entire image captured by the camera. 

[0060] Alternatively, the camera captures images of areas larger than can be 
displayed and only a portion of the image is displayed. This is considered less preferable 
since it increases the bandwidth requirements and often for no reason as the additional 
data is not displayed. 
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[0061] Advantageously, when implemented with independent position indicators for 
each of the HMD 1 and the camera 1 1 and independent from the mechanism 13 for 
moving the camera, all types of motion are compensated for including inaccuracies of the 
mechanism 13, delays induced by communications, delays induced by the mechanism 13, 
processing delays, fine operator motions and so forth. 

[0062] When processing is done local to the HMD or on a computer at a same 
location with minimal delays therebetween, each image is accurately aligned on the 
display within a time delay error related only to the processing and the delay in reading 
HMD sensor data. 

[0063] Thus, discontinuous scene changes are changed into smooth transitions in 
accordance with the expected visual result. 

[0064] It is also within the scope of the invention to process the image data prior to 
display thereof in order to determine features or locations within the image data to 
highlight or indicate within the displayed image. For example, contrast may be improved 
for generally light or dark images. Also, features may be identified and labeled or 
highlighted. Alternatively, icons or other images are superimposed on the displayed 
image without processing thereof. 

[0065] Alternatively, the control values are determined in the mechanism for pointing 
the camera instead of by the first computer. In such an embodiment, the HMD position 
data is transmitted to the remote system wherein a camera movement related to the HMD 
movement is determined and initiated. 

[0066] The above described embodiment compensates for orientation - motion about 
any of three rotational axes. Alternatively, the invention compensates for displacement - 
linear motion along an axis. Further alternatively, the invention compensates for both 
linear motion and motion about any of the rotational axes. Displacement and orientation 
are both forms of position and data relating to one or both is referred to here and in the 
claims, which follow, as position data. 

[0067] The above described embodiment does not correct images for perspective 
distortion. Doing so is feasible within the concept of time/motion compensation according 
to the invention, however it is not generally applicable to use with a single camera, since 



13 



Doc. No. 50412-01 US/PCT 

the depth of field of the observed scene varies. It would require capturing of depth data 
using a range sensor or a three-dimensional vision system. 

[0068] Though the above embodiment is described with reference to a physical 
communication link or a wireless communication link between different components, 
clearly, either is useful with the invention so long as it is practicable. Also, though the 
HMD is described as a computer peripheral, it could be provided with an internal 
processor and act as a stand alone device. 

[0069] According to another embodiment of the invention, areas within the field of 
view that do not correspond to displayed image locations are filled with current image 
data relating to earlier captured images for those locations. Preferably, any earlier 
captured images are deemphasized within the field of view in order to prevent the 
operator from being confused by "stale" image data. For example, each image received 
from the camera is buffered with its associated position data. When some areas within the 
field of view are not occupied by image data, the processor determines another image 
having image data for those locations within the field of view, the locations determined in 
accordance with the transform performed based on the camera position data associated 
with the earlier captured image and with the current HMD position data. The image data 
is then displayed at the determined location(s) in a "transparent" fashion. For example, it 
may be displayed with a lower contrast appearing almost ghostlike. Alternatively, the 
colours are faded to provide this more ghostlike appearance. Further alternatively, it is 
displayed identically to the current image data. 

[0070] The above description is by way of example and is not intended to limit the 
forgoing claims. 
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Claims 

1. A method of motion compensation for head mounted displays comprising the 
steps of: 

providing a captured image from an image capture device to a head mounted 
display including a display having a field of view; 

providing camera position data relating to a position of the camera and associated 
with the captured image; 

providing HMD position data relating to a position of the head mounted display; 

transforming the image to vary a displayed location of static objects within the 
image relative to the field of view in accordance with the camera position data and the 
HMD position data; and, 

displaying portions of the image at the displayed locations, those portions 
remaining within the field of view. 

2. A method according to claim 1, wherein the HMD position data is data relating to 
the present position of the head mounted display and wherein the camera position data is 
data relating to the position of the camera when the associated image is captured and 
wherein the step of transforming the image includes the steps of: 

determining an offset between the HMD position and the camera position; and, 
offsetting the image such that it is offset an amount proportional to the offset 
between the HMD position and the camera position. 

3. A method according to claim 2, wherein portions of the field of view for which 
image data is unavailable are filled with a predetermined fill. 

4. A method according to claim 3, wherein the predetermined fill has features for 
maintaining orientation of a wearer of the head mounted display. 

5. A method according to claim 1, wherein the captured image is larger than the 
image necessary to fill the field of view of the head mounted display, and wherein only a 
portion of the captured image is displayed. 
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6. A method according to claim 1, wherein the system comprises two head mounted 
displays and two cameras. 

7. A method according to claim 1, wherein the system comprises: 

an independent position sensor for sensing the camera position and for providing 
the camera position data. 

8. A method according to claim 7, wherein the system comprises: 

an independent position sensor for sensing the head mounted display position and 
for providing the HMD position data. 

9. A method according to claim 1 , wherein the system comprises: 

an independent position sensor for sensing the head mounted display position and 
for providing the HMD position data. 

1 0. A method according to claim 1 , wherein the system comprises: 
a mechanism for moving the camera; and 

means for transmitting the HMD position data to a system in communication with 
the mechanism for moving the camera, 

wherein the mechanism for moving the camera moves the camera in response to a 
change in HMD position data. 

11. A method according to claim 1, wherein the HMD position data comprises 
orientation data. 

12. A method according to claim 11, wherein the camera position data comprises 
orientation data. 

13. A method according to claim 12, wherein the camera position data comprises 
displacement data. 

14. A method according to claim 1, wherein the HMD position data comprises 
displacement data. 
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15. A method according to claim 14, wherein the camera position data comprises 
displacement data. 

16. A method according to claim 15, wherein the camera position data comprises 
orientation data. 

17. A method according to claim 1, comprising the step of transforming the image to 
reduce perspective distortion, wherein the camera includes a range sensor. 

18. A motion compensation apparatus for head mounted displays comprising: 

a head mounted display for displaying portions of an image at displayed locations, 
those portions remaining within the field of view; 

an image capture device for providing a captured image to a head mounted display 
including a monitor having a field of view; 

a sensor for providing camera position data relating to a position of the camera 
and associated with the captured image; 

a sensor for providing HMD position data relating to a position of the head 
mounted display; and 

a processor for transforming the image to vary a displayed location of static 
objects within the image relative to the field of view in accordance with the camera 
position data and the HMD position data. 

19. A motion compensation apparatus for head mounted displays according to 
claim 18, wherein the image capture device comprises a range sensor. 

20. A motion compensation apparatus for head mounted displays according to 
claim 19, wherein the range sensor includes a stereoscopic imaging system. 

21. A method according to claim 2 comprising the step of: 

when portions of the field of view for which image data is unavailable are 
detected, determining earlier captured image data that when transformed in dependence 
upon its associated camera position data and the HMD position data is for display within 
the portions; and, 
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displaying with those portions of the field of view the transformed earlier captured 
image data. 

22. A method according to claim 21, wherein the step of displaying transformed 
earlier captured image data includes a step of de-emphasising the transformed earlier 
captured image data. 
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Abstract of the Disclosure 

A method and system for time-motion compensation for use with head mounted 
displays is disclosed. According to the method, a remote camera captures an image for 
display on a head-mounted display (HMD) including a viewing window. The image and 
camera position data is transmitted to a system including the HMD for display to a wearer 
of the HMD. The HMD position is determined. An offset between the HMD position and 
a known position of the HMD is determined as is an offset between the camera position 
and a known position of the camera. The image is offset relative to the viewing window 
of the image based on the difference between the two determined offsets. 
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Method and System for Time/motion Compensation for Head Mounted Displays 




The present invention generally relates to telepresence systems and more 
particularly relates to motion compensation in telepresence systems. 

5 Background of the Invention 

The field of remote control has come a long way since the days of watching a 
model aircraft fly under the control of a handheld controller. Robotics and remote robotic 
manipulation have created a strong and pressing need for more remote and better remote 
control systems. Obviously, an ideal form of remote control involves providing an 
10 operator with all the sensations of operating the remote robot without the inherent 
dangers, travel, and so forth. In order to achieve this, a telepresence system is used. 



mounted display. The system provides visual feedback from a remote location to an 



;; ? 15 operator. For example, in a telepresence system for an automobile, the front windshield is 



provided with a camera. The controls of the vehicle are provided with actuators for 
automatically manipulating same. An operator is provided with a duplicate of the cabin of 
the car. The windshield is replaced with a display and the controls are linked via 
communications to the actuators within the vehicle. Turning of the steering wheel in the 



ry 20 cabin of the car causes the steering wheel to turn in the vehicle. Similarly, the camera 
captures images in front of the car and they are displayed on the display in the cabin of 
the car. 

Presently, there is a trend toward providing the visual feedback using a head 
mounted display (HMD). A head mounted display is a small display or two small displays 

25 mounted for being worn on a users head. Advantageously, an HMD with two displays 
provides stereo imaging allowing a user to perceive depth of field. Alternatively, such an 
HMD provides two identical images, one to each display. Unfortunately, the head 
mounted display only presents a user with information from approximately in front of the 
user. Thus, when a user turns their head, the image seen and the expected image differ. 

30 Therefore, the camera is mounted on a mechanism which moves in accordance with 



m 
a; 



Telepresence systems are sensory feedback systems for allowing sensing and 
monitoring of remote systems. A typical telepresence sensor is a camera and a head 
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detected HMD movement. Thus, the image before the user is in accordance with the 
user's head position. 



Generally, it is an object of telepresence systems to provide a visual sensation of 
being in the place of the robot and a control system for controlling the robot as well. 
5 Thus, telepresence systems aim to provide feedback that is appropriate to different 
situations. 

Unfortunately, a camera does not move in exact synchronisation with the HMD so 
the image is not perfectly aligned with the expectations of the user during head motion. 
This misalignment can result in disorientation and nausea on the part of an operator. 

1 0 The disclosures in U.S. Patent 5,579,026 issued on Nov. 26, 1996 in the name of 

Tabata and in U.S. Patent 5,917,460 issued on Jun. 29, 1999 in the name of Kodama 
focus on image display for use in, for example, virtual reality and games. In there is 
described a head mounted display in which the position of the projected image can be 
displaced in response to a control unit or in response to the rotational motion of the 

1 5 operator's head. The essence of the head-tracking implementation is that from the user's 
perspective, the image can be made to remain substantially stationary in space during 
head movements, by being manipulated in a manner opposite to the movements. 
Significantly, the patents do not relate to visual telepresence using slaved cameras. In the 
slaved camera implementation, the camera should follow the motion of the HMD and, as 

20 such, compensation for HMD motion is unnecessary since the image is always of a 
direction in which the head is directed. 

Further because U.S. Patent 5,579,026 relates to displaying a simulated planar 
image, such as a simulation of a television screen located in virtual space in front of the 
user, the patent provides for a fixed frame of reference relative to a wearer of the HMD. 
25 The images in any direction are simulated thus being formed as needed. Unfortunately, in 
telepresence systems, often the video data relating to a particular direction of view is 
unavailable. This complicates the system significantly and as such, the prior art relating to 
video data display is not truly applicable and, one of skill in the art would not refer to 
such. 

30 In U.S. Patent 5,9 1 7,460 issued on Jun. 29, 1 999 in the name of Kodama a system 

addressing the three-axes displacement (up/down, left/right, frontwards/backwards) of a 

2 
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HMD is provided. The displacement appears to linear and is accommodated through a 
mechanical mechanism. The displays are moved in response to detected movement of a 
head and as such, objects remain somewhat stationary from the visual perspective of the 
user. 

It is not well suited to use in telepresence wherein a camera tracks the motion of 
the HMD. One of skill in the art, absent hindsight, would not be drawn to maintaining a 
visual reference when a head is turned, for a telepresence system wherein a camera is 
rotated in response to head movement. Of course, the different problem results in a 
different solution. 

For example, in telepresence systems, the delay between camera image capture 
and head motion is often indeterminate. It is not a workable solution to implement the 
system of the above referenced patents to solve this problem. Because of the unknown 
delays caused by camera response time and communication delays, the solution is not 
trivial. 

In U.S. Patent 5,933,125 a system is disclosed using prediction of the head 
movement to pre-compensate for the delay expected in the generation of a virtual image, 
nominally in a simulated environment. By this means, a time lag in the generation of 
imagery is compensated for by shifting the scene to provide a stable visual frame of 
reference. This method is applicable to short delays and small displacements, where head 
tracking information can be used to predict the next head position with reasonable 
accuracy. The patent discloses 100msec as a normal value. Effective prediction of head 
motion is aided by comprehensive information about head movement, including angular 
head velocity and angular acceleration. For small head movements, errors induced are 
small. Typically, these occur in a small period of time. The disclosed embodiments rely 
on knowledge of the time delay, which is nominally considered to be constant. 
Unfortunately, when the time delays grow large allowing for substantial motion of a head, 
the errors in the predictive algorithm are unknown and the system is somewhat 
unworkable. 

Furthermore, US 5,933,125 cannot compensate for unanticipated image 
movement, only that which occurs in correct response to the operator's head movement. 
Also, it does not relate to visual telepresence systems using remote slave cameras. 
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It would be highly advantageous to provide a system that does not rely on any 
form of prediction for compensation and which works with variable delays between 
image capture and image display. 

5 Object of the Invention 

In order to overcome these and other shortcomings of the prior art, it is an object 
of the invention to provide a method of compensating for time delays between head 
motion and camera motion in telepresence systems. 

Summary of the Invention 

10 The invention relates to a method and apparatus that provides a wearer of an 

HMD with a stable frame of visual reference in cases where there may be time delays or 

M> unwanted motion within the visual capture/visual display systems. 

O 

According to the invention, in order to eliminate some of the disorientation caused 
by time delays in camera motion when a head motion occurs, an image shown on the 
e 15 display of a head-mounted display (HMD) is offset relative to the field of view of the 
HMD until the camera position is again synchronised with the HMD position. Offsetting 
p of the image results in areas of the display for which no image information is available. 

J! These display areas are provided fill data in the form of a solid shading or some feature 

ft) set for providing visual cues. When the transformed images again overlap the display, the 

=' 20 fill is no longer necessary. 

In accordance with the invention there is provided a method of motion 
compensation for head mounted displays. The method includes the following steps: 
providing an image from an image capture device to a head mounted display including a 
monitor having a field of view; providing camera position data associated with the image; 
25 providing head position data; adjusting the image location relative to the field of view of 
the monitor in accordance with the camera position data and the head position data; and, 
displaying portions of the image at the adjusted locations, those portions remaining within 
the field of view. 

Typically position data includes at least one of orientation data and location data. 
30 Location data is also referred to as displacement data. Typically, portions of the field of 
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view without image data are filled with a predetermined fill. When none of the image data 
is for display within the field of view, the entire field of view is filled with the 
predetermined fill. 

For example, the image is adjusted by the following steps: determining an offset 
5 between the head mounted display position and the camera position; and, offsetting the 
image such that it is offset an amount equal to the offset between the head mounted 
display position and the camera position. 

Advantageously, such a system is not limited by the accuracy of a predictive 
process nor by the time delay between image capture and image display. Instead, it is 
1 0 reactive, and uses sensed information on HMD position and camera position to formulate 
a transformation for the captured image. The present invention has no limit to the time 
delays for which compensation is possible since the required head position information 
and camera position information are sensed at different times allowing compensation for 
any delay between sensing one and then sensing the other. 

15 Further advantageously, the present invention requires no knowledge of the time 

delay in the system and functions properly in the presence of non-constant time delays. 
There is no requirement that the time delay be measured and it is not used in determining 
the transform of the image. 

Brief Description of the Drawings 

20 The invention will now be described in conjunction with the drawings in which: 

Fig. 1 is a simplified block diagram of a system incorporating an HMD coupled to a first 
computer and in communication with a remote camera; 

Figgis a simplified diagram showing axes of movement of the systems involved: 

Fig. 3 a is a simplified diagram showing a simulated view of an image appearing within 



25 the field of view of an HMD; 

JFjg^3b is a simplified diagram showing a simulated view of a portion of the image of Fig. 
3a offset vertically within the field of view of an HMD in response to a downward motion 
of a user's head; 
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F|g- 3 c is a simplified diagram showing a simulated view of a portion of the image of Fig. 
3a offset horizontally within the field of view of an HMD in response to a lateral motion 
of a user's head; 

Fig. 3d is a simplified diagram showing a simulated view of a portion of the image of Fig. 
5 3a tilted within the field of view of an HMD in response to a tilting motion of a user's 
head; 

Fig. 3 e is a simplified diagram showing a simulated view of a portion of the image of Fig. 
3a tilted and offset both vertically and horizontally within the field of view of an HMD in 
response to a tilting motion combined with a lateral and a horizontal motion of a user's 
10 head; 

Fjg^_4ajs a simplified diagram showing a simulated view of an image appearing within 
the field of view of an HMD; 

Fig. 4b is a simplified diagram showing a simulated view of a portion of the image of Fig. 
4a offset horizontally within the field of view of an HMD in response to a lateral motion 
15 of a user's head; 

Fj g. 4c is a simplified diagram showing a simulated view of an image appearing within 
the field of view of an HMD including a portion of the image of Fig. 4a as well as 
additional image data captured and displayed within the field of view of an HMD when 
the camera motion is partially caught up with the lateral motion of a user's head; 

20 JFig^4d is a simplified diagram showing a simulated view of an image appearing within 
the field of view of an HMD including a portion of the image of Fig. 4a as well as 
additional image data captured and displayed within the field of view of an HMD after 
the camera motion is fully caught up with the lateral motion of a user's head; 

_ Fig. 5 i sa simplified block diagram of a system incorporating an HMD coupled to a first 
25 computer and in communication across a network with a second computer coupled to a 
remote camera; 

Fig. 6 is a simplified flow diagram of a method according to the invention; 
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JFig . 7 , is a simplified block diagram of a telepresence system communicating via a 
satellite communications link; 



Fig. 8 is an image captured by a remote camera as captured; 

Fig. 9 is an image of the same image as that of fig. 8 transformed within an endless image 
5 display space; and, 

Fi& 10 is an image of the same image as that of fig. 9 as displayed on a display having a 
finite image display space. 

Detailed Description of the Invention 

The present invention is described with reference to telepresence systems 
10 operating over long distances such that significant time delays occur between head 
motion and image display of an image for a current head orientation. It is, however, 
equally applicable when a camera drive mechanism provides insufficient response rate to 
allow comfortable viewing of images during normal head motion. It is also applicable in 
situations where unwanted and unmodeled motion of the camera is possible, such as when 
1 5 the camera is mounted on a moving platform. 

Referring to Fig. 1, a simplified block diagram of a telepresence system is shown. 
A head mounted display (HMD) 1 including a three-axes head tracker 3 is worn by an 
operator 5. The HMD 1 is coupled with a first computer 7 and provides to the first 
computer 7 HMD values for the HMD position in the form of pitch 21, yaw 22, and roll 

20 23 angles of the HMD 1 as shown in Fig. 2. Of course, since the HMD 1 is being worn by 
an operator 5, these HMD values relate to the head position of the operator 5. These 
values are provided to the first computer 7 at intervals, preferably more than 1 00 times 
each second, though other intervals are also acceptable. The HMD values are converted 
by the first computer 7 into control values for controlling positioning of a camera 1 1. The 

25 control values are transmitted to a mechanism 13 for pointing the camera in order to 
affect camera orientation. As is seen in Figure 2, the mechanism controls pitch 24, yaw 25 
and roll 26 of the camera 1 1. In response to the control values, the camera position moves 
in accordance with the movement of the HMD 1 . 
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When the mechanism 13 for pointing the camera is physically coupled to the first 
computer 7, the camera 11 begins to move when HMD motion is detected. The lag 
between camera motion and HMD motion is determined by communication delays, which 
are very small, processing delays, which may be minimal, and pointing mechanism 
performance, which varies. These delays often result in an image provided from the 
camera 1 1 remaining static while the HMD 1 is in motion or moving substantially slower 
than the HMD motion. Of course, since the operator's mind expects motion within the 
visual frame, this is disconcerting and often results in nausea and disorientation. 

This problem is even more notable when communication delay times are 
significant such as when used for terrestrial control of systems in space. There, the delay 
is in the order of seconds and, as such, the disorientation of an operator during HMD 
motion is significant. Significantly, disorientation is a cause of operator fatigue resulting 
in limited operator use of a system or limited use of a system during a day. 

Referring again to Fig. 1, the camera 1 1 is constantly acquiring images at a video 
data capture rate. Each image is transmitted to the first computer for processing, if 
required, and for provision to the HMD 1. According to the invention, the remote system 
also provides camera position information to the first computer 7 and associated with 
each image. Thus, each frame received by the first computer 7 has associated therewith 
camera position information. The camera position information is preferably relative to a 
known orientation. Alternatively, it is transformed by the first computer 7 into position 
information relative to a known camera orientation and in a coordinate space analogous to 
that of the HMD 1. 

The HMD position values are used to determine a current HMD orientation in a 
coordinate space analogous to that of the camera 1 1. As such, an offset between camera 
orientation and HMD orientation is determinable. Since the HMD 1 is being worn by an 
operator 5 the HMD orientation is directly correlated to the position of the head of the 
operator 5. Of course, the direct correlation is related to sensed position data and in use is 
generally an approximate direct correlation due to a refresh rate of the HMD position 
sensor. The offset between the camera orientation and the HMD orientation is related to a 
delay between the local system and the remote system. 
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Therefore when a non-zero offset is determined, the first computer offsets the 
image provided by the camera relative to the field of view of the HMD in order to 
compensate for the determined offset Referring to Figs. 3a, 3b, 3c, 3d and 3e, some 
examples of image displays are shown. In Fig. 3a, the image is shown for a zero offset 
5 between camera orientation and HMD orientation. This is the steady state of the feedback 
system since the HMD 1 is directed in a same direction as the camera 1 1 and the image 
displayed on the display within the HMD 1 is the same as the image captured by the 
camera. When the HMD orientation is angled down from the camera position, the image 
is offset in a vertical direction as shown in Fig. 3b. When the camera orientation is offset 
10 horizontally, the image is offset horizontally as shown in Fig. 3c. In Figs. 3d and Fig. 3e, 
the image is rotated and offset relative to the field of view because the camera orientation 
is rotated and offset relative to the HMD orientation. 

Though the described instantaneous corrections shown in Figs. 4a, 4b, 4c and 4d 
appear simple, the steady state nature of the system requires an ever changing imaging 
15 perspective and display perspective. Thus a comparison is necessary between two 
dynamic sets of sensed position data. 

Referring to Figs. 4a, 4b, 4c and 4d, field of view is shown for the HMD 1 during 
a left turn of the operator's head. At first (before the head turn) in a steady state, the exact 
image captured by the camera 1 1 is shown in the display of the HMD 1 at Fig. 4a. When 

20 the head turns, the operator 5 "expects" the image to move to the right since the image is 
not part of the operator 5 and is within their field of view. This expectation is either 
conscious or unconscious. Imagining that the image remains static as the HMD moves, it 
is clear that disorientation would result since individuals take cues from their visual field 
of view during head movement. In order to provide the operator 5 with the "expected" 

25 displacement of objects in the image, the image is offset to a location approximately the 
same as the orientation difference between the HMD 1 and the camera 1 1. For example, 
in the image of Fig. 4b the lighthouse is shifted out of the field of view by the rotation of 
the head. Turning the HMD a degrees, an operator expects static objects within the field 
of view, such as a lighthouse, to shift a degrees within the field of view. This is important 

30 to maintaining comfort of the operator in their personal vision system (their eyes and their 
mind). At a same time, the camera begins to move to match its orientation to that of the 
HMD. Thus, as shown in the image of Fig. 4c more of the scene within the operator's 

9 
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field of view is now available from the camera 1 1. As the camera orientation "catches up" 
with the HMD orientation, the field of view of the camera and that of the HMD overlap 
more. When the camera 11 is "caught up," the field of view of the HMD again shows an 
entire image captured by the camera as shown in the image of Fig. 4d. 

Referring to Fig. 5, another embodiment of the invention is shown for use on a 
network. Here for example, two computers 7 and 15 communicate via a network or 
networks 17. The first computer 7 includes the HMD 1 as a peripheral thereof. The 
second computer 15 includes the camera 1 1 and mechanism 13 for pointing the camera as 
peripherals thereof. Here the processing is performed in either of the first computer 7 or 
the second computer 15 though the image processing is preferably performed in the first 
computer 7 in case of network delays that could cause image offset and result in 
disorientation of an operator 5. Of course, when network delays are known to be 
significant, it is important that image processing is performed on the first computer. 

Referring to Fig. 6, a simplified flow diagram of a method of performing the 
invention is shown. An image is captured by the camera 1 1 . A sensor captures position 
data in the form of camera orientation values for pitch, roll and yaw. The position data is 
preferably captured concurrently with the image. Alternatively, it is captured 
approximately at a same time but offset by a finite amount either immediately after image 
capture or immediately before. The position data is then associated with the image data. A 
simple method of associating the data is by encoding the position data with the image 
data either as header or trailer information. Of course, the image and the position data 
could also be identified with an associating identifier such as an image frame number. 
Alternatively, the two data are transmitted in parallel in a synchronous environment. 

The image and position data are then transmitted to the first computer 7. When the 
image and position data are received, they are prepared for processing at the first 
computer 7. Then, the position data of the HMD 1 is acquired by the first computer 7 and 
is used to transform the image in accordance with the invention. The transformed image 
is provided to the display and is displayed thereon to the operator 5. Because the HMD 
position data is gathered immediately before it is needed, the delay between HMD 
position data capture and display of the transformed image is very small and results in 
little or no operator disorientation. 



10 



SUBSTITUTE SHEET (RULE 26) 



WO 01/22149 PCT/CA00/01063 
Concurrently, position data is provided to the mechanism 13 at intervals and the 
mechanism moves the camera 1 1 in accordance with received position data and a current 
orientation of the camera 1 1 . 

Typically, the step of transforming the image comprises the following steps, some 
5 of which are performed in advance. A correlation between angular movement and display 
or image pixels is determined such that an offset of a degrees results in displacement of 
the image by N pixels in a first direction and by M pixels in a second other direction. A 
transform for rotating the image based on rotations is also determined. Preferably, the 
transforms are sufficiently simple to provide fast image processing. That said, a small 
1 0 image processing delay, because it forms substantially the delay in displaying the data, is 
acceptable. 

Once the image data is received, it is stored in memory for fast processing thereof. 
The HMD position data is acquired and is compared to the camera position data. The 
difference is used in performing the transform to correct the image position for any HMD 
15 motion unaccounted for by the mechanism 13, as of yet. Also, the method corrects for 
unintentional movements of the camera 11 when the camera position sensor is 
independent of the mechanism 13, for example with an inertial position sensor. 

In the above embodiment, a general purpose processor is used to transform the 
image. In an alternative embodiment, a hardware implementation of the transform is used. 
20 A hardware implementation is less easily modified, but has a tremendous impact on 
performance. Using parallel hardware transformation processors, an image can be 
transformed in a small fraction of the time necessary for performing a software 
transformation of the image. 

Referring to Fig. 7 a satellite based telepresence system is shown. Here delays in 
25 the order of seconds between head motion and image display result. Further, the delays 
are not always predictable. Here an HMD 101 is shown positioned on the head of an 
operator 5. The HMD is provided with a head tracker 103 for sensing position data 
relative to the HMD. The HMD is also coupled with a computer 107 for providing display 
data to the HMD and for providing the HMD position data to a communications link 108. 
30 The communications link 108 uplinks the HMD position data to a satellite 117 from 
which it is transmitted to a transceiver 116. A computer 115 in communication with the 

11 
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transceiver provides the data to a gimbal 113 for repositioning a camera 111 in 
accordance therewith. The camera 111 captures images which are provided along a 
reverse communication path - computer 115, transceiver 116, satellite 117, 
communications link 108 - to the computer 107. Optionally, a different return path is 
5 used. There the image data is processed for display within the HMD 101. With the 
images, camera position data sensed by a sensor 114 is also provided. The camera 
position data is associated with an image or images captured at approximately a time the 
camera 1 1 1 was in that sensed position. 

The computer 107 uses the camera position data and the image along with data 
10 received from the head tracker 103 to transform the image in accordance with the 
invention. As is evident, the delay between HMD motion and camera motion is 
measurable in seconds. The delay between camera image capture and receipt of the image 
at the computer 107 is also measurable in seconds. As such, significant disorientation of 
the user results absent application of the present invention. 

15 Referring to Fig. 8, an image captured by the camera 1 1 1 is shown. The image is 

displayed as captured when the HMD and the camera orientations are aligned, the camera 
orientation at a time of image capture and the HMD orientation at a time of display. If the 
orientations are offset one from another, the image is shifted within the field of view of 
the operator as shown in Fig. 9. Since there is no image data beyond the camera imaging 

20 field of view, the remainder of the display area is shaded with a neutral colour such as 
gray. Alternatively, the remainder of the display area is shaded to provide reference 
points to further limit disorientation of the operator 105. Further alternatively, the portion 
of the field of view for which no image data is available is left blank. Typically blank 
areas are black in order not to distract an operator. Referring to Fig. 10, when the camera 

25 111 orientation is "caught up" with the HMD orientation, the field of view of the HMD 
again shows an entire image captured by the camera. 

Alternatively, the camera captures images of areas larger than can be displayed 
and only a portion of the image is displayed. This is considered less preferable since it 
increases the bandwidth requirements and often for no reason as the additional data is not 
30 displayed. 
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Advantageously, when implemented with independent position indicators for each 
of the HMD 1 and the camera 1 1 and independent from the mechanism 13 for moving the 
camera, all types of motion are compensated for including inaccuracies of the mechanism 
13, delays induced by communications, delays induced by the mechanism 13, processing 
5 delays, fine operator motions and so forth. 

When processing is done local to the HMD or on a computer at a same location 
with minimal delays therebetween, each image is accurately aligned on the display within 
a time delay error related only to the processing and the delay in reading HMD sensor 
data. 

10 Thus, discontinuous scene changes are changed into smooth transitions in 

accordance with the expected visual result. 

M It is also within the scope of the invention to process the image data prior to 

display thereof in order to determine features or locations within the image data to 
03 highlight or indicate within the displayed image. For example, contrast may be improved 

~~ 15 for generally light or dark images. Also, features may be identified and labeled or 
£ highlighted. Alternatively, icons or other images are superimposed on the displayed 

B image without processing thereof. 

t Alternatively, the control values are determined in the mechanism for pointing the 

camera instead of by the first computer. In such an embodiment, the HMD position data 

S- 20 is transmitted to the remote system wherein a camera movement related to the HMD 

iw 

movement is determined and initiated. 

The above described embodiment compensates for orientation - motion about any 
of three rotational axes. Alternatively, the invention compensates for displacement - 
linear motion along an axis. Further alternatively, the invention compensates for both 
25 linear motion and motion about any of the rotational axes. Displacement and orientation 
are both forms of position and data relating to one or both is referred to here and in the 
claims, which follow, as position data. 

The above described embodiment does not correct images for perspective 
distortion. Doing so is feasible within the concept of time/motion compensation according 
30 to the invention, however it is not generally applicable to use with a single camera, since 

13 
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the depth of field of the observed scene varies. It would require capturing of depth data 
using a range sensor or a three-dimensional vision system. 



Though the above embodiment is described with reference to a physical 
communication link or a wireless communication link between different components, 
5 clearly, either is useful with the invention so long as it is practicable. Also, though the 
HMD is described as a computer peripheral, it could be provided with an internal 
processor and act as a stand alone device. 

According to another embodiment of the invention, areas within the field of view 
that do not correspond to displayed image locations are filled with current image data 
10 relating to earlier captured images for those locations. Preferably, any earlier captured 
images are deemphasized within the field of view in order to prevent the operator from 
being confused by "stale" image data. For example, each image received from the camera 
t is buffered with its associated position data. When some areas within the field of view are 
p not occupied by image data, the processor determines another image having image data 

1 5 for those locations within the field of view, the locations determined in accordance with 

m 

"-4 the transform performed based on the camera position data associated with the earlier 

Jz 

captured image and with the current HMD position data. The image data is then displayed 
at the determined locations) in a "transparent" fashion. For example, it may be displayed 

hy with a lower contrast appearing almost ghostlike. Alternatively, the colours are faded to 

20 provide this more ghostlike appearance. Further alternatively, it is displayed identically to 

C the current image data. 

The above description is by way of example and is not intended to limit the 
forgoing claims. 
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1. A method of motion compensation for head mounted displays comprising the steps of: 
providing a captured image from an image capture device to a head mounted display 
5 including a display having a field of view; 

providing camera position data relating to a position of the camera and associated with 
the captured image; 

providing HMD position data relating to a position of the head mounted display; 
transforming the image to vary a displayed location of static objects within the image 
10 relative to the field of view in accordance with the camera position data and the HMD 
position data; and, 

displaying portions of the image at the displayed locations, those portions remaining 
within the field of view. 

15 2. A method according to claim 1 wherein the HMD position data is data relating to the 
present position of the head mounted display and wherein the camera position data is data 
relating to the position of the camera when the associated image is captured and wherein 
the step of transforming the image includes the steps of: 
determining an offset between the HMD position and the camera position; and, 

20 offsetting the image such mat it is offset an amount proportional to the offset between the 
HMD position and the camera position. 

3. A method according to claim 2 wherein portions of the field of view for which image 
data is unavailable are filled with a predetermined fill. 

25 

4. A method according to claim 3 wherein the predetermined fill has features for 
maintaining orientation of a wearer of the head mounted display. 

5. A method according to claim 1 wherein the captured image is larger than the image 
30 necessary to fill the field of view of the head mounted display, and wherein only a portion 

of the captured image is displayed. 

6. A method according to claim 1 wherein the system comprises two head mounted 
displays and two cameras. 

15 
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7. A method according to claim 1 wherein the system comprises: 

an independent position sensor for sensing the camera position and for providing the 

camera position data. 

5 8. A method according to claim 7 wherein the system comprises: 

an independent position sensor for sensing the head mounted display position and for 
providing the HMD position data. 

9. A method according to claim 1 wherein the system comprises: 

an independent position sensor for sensing the head mounted display position and for 
10 providing the HMD position data. 

10. A method according to claim 1 wherein the system comprises: 
a mechanism for moving the camera; and 

means for transmitting the HMD position data to a system in communication with the 
mechanism for moving the camera, 
15 wherein the mechanism for moving the camera moves the camera in response to a change 
in HMD position data. 

1 1. A method according to claim 1 wherein the HMD position data comprises orientation 
data. 

20 12. A method according to claim 11 wherein the camera position data comprises 
orientation data. 

13. A method according to claim 12 wherein the camera position data comprises 
displacement data. 

25 

14. A method according to claim 1 wherein the HMD position data comprises 
displacement data. 

15. A method according to claim 14 wherein the camera position data comprises 
30 displacement data. 
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16. A method according to claim 15 wherein the camera position data comprises 
orientation data. 



17. A method according to claim 1 comprising the step of transforming the image to 
5 reduce perspective distortion, wherein the camera includes a range sensor. 

18. A motion compensation apparatus for head mounted displays comprising: 

a head mounted display for displaying portions of an image at displayed locations, those 
portions remaining within the field of view; 
10 an image capture device for providing a captured image to a head mounted display 
including a monitor having a field of view; 

a sensor for providing camera position data relating to a position of the camera and 
associated with the captured image; 

a sensor for providing HMD position data relating to a position of the head mounted 
15 display; and 

a processor for transforming the image to vary a displayed location of static objects 
within the image relative to the field of view in accordance with the camera position data 
and the HMD position data. 

20 19. A motion compensation apparatus for head mounted displays according to claim 18 
wherein the image capture device comprises a range sensor. 

20. A motion compensation apparatus for head mounted displays according to claim 19 
wherein the range sensor includes a stereoscopic imaging system. 

25 

21. A method according to claim 2 comprising the step of: 

when portions of the field of view for which image data is unavailable are detected, 
determining earlier captured image data that when transformed in dependence upon its 
associated camera position data and the HMD position data is for display within the 
30 portions; and, 

displaying with those portions of the field of view the transformed earlier captured image 
data. 



17 



SUBSTITUTE SHEET (RULE 26) 



WO 01/22149 PCT/CA00/01O63 

22. A method according to claim 21 wherein the step of displaying transformed earlier 
captured image data includes a step of de-emphasising the transformed earlier captured 
image data. 
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